Producing photorealistic images or cartoon images from a 3D scene model requires computing the light transport through a scene, which requires estimation of a complex multidimensional integral of the light scene function at every pixel of the image. For example, generating effects like depth of field and motion blur requires integrating over domains such as lens/viewpoint position and time. Monte Carlo (MC) rendering systems approximate this integral by tracing light rays (samples) in the multidimensional space to evaluate the scene function. Although an approximation to this integral can be quickly evaluated with just a few samples, the error of this estimate relative to the true value appears as unacceptable noise in the resulting image. Since the variance of the MC estimator decreases linearly with the number of samples, many samples are required to get a reliable estimate of the integral. The high cost of computing additional rays results in lengthy render times that negatively affect the applicability of MC renderers in modern film production.
One way to mitigate this problem is to quickly render a noisy image with a few samples and then filter it as a post-process to generate an acceptable, denoised result. This approach has been the subject of extensive research in recent years. Successful methods typically use feature-based filters (e.g., cross-bilateral or cross non-local means filters) to leverage additional scene features, such as world position, that help guide the filtering process. Since these features are highly correlated with scene detail, using them in the filtering process greatly improves the quality of the results.
Some approaches have used this information to handle specific distributed effects such as global illumination and depth of field. However, a major challenge is how to exploit this additional information to denoise distributed effects, which requires setting the filter parameters for all features so that noise is removed while scene detail is preserved. To do this, some have proposed to use the functional dependencies between scene features and random parameters calculated using mutual information, a process that removes noise but was slow. Several other algorithms build upon this by using error estimation metrics to select the best filter parameters from a discrete set. The main drawback of these methods is that their error metrics are usually noisy at low sampling rates, reducing the accuracy of filter selection. Furthermore, they choose the filter parameters from a preselected, discrete set that may not contain the optimum. As a result, these methods produce images with over/under blurred regions.
Since the introduction of distributed ray tracing by Cook et al., (Cook et al., “Distributed Ray Tracing” Computer Graphics (Proceeding of SIGGRAPH 84) 18, 3 (1984), 137-144) researchers have proposed a variety of algorithms to address the noise in Monte Carlo (MC) rendering. Some of these include variance reduction techniques, low-discrepancy sampling patterns, new Monte Carlo formulations with faster convergence, and methods that exploit specific properties of the multidimensional integrand position or reuse samples based on the shape of the multidimensional integrand.
Filtering approaches render a noisy image with a few samples and then denoise images through a filtering process. Some methods adaptively sample as well, further improving the results. Some previous work on MC filtering use only sample color during filtering and others use additional scene information.
Color-based filter methods are inspired by traditional image denoising techniques and use only pixel color information from the rendering system to remove MC noise. Early work by Lee and Redner used nonlinear filters (median and alpha-trimmed mean filters) to remove spikes while preserving edges. See, Lee & Redner, “A note on the use of nonlinear filtering in computer graphics,” IEEE Computer Graphics and Applications 10, 3 (May 1990), 23-29. Rushmeier and Ward proposed to spread the energy of input samples through variable width filter kernels. See, Rushmeier and Ward, “Energy preserving non-linear filters,” ACMSIGGRAPH '94 (New York, N.Y., USA, 1994), pp. 131-138. To reduce the noise in path-traced images, Jensen and Christensen separated illumination into direct and indirect components, filtered the indirect portion, and then added the components back together. See, Jensen and Christensen, “Optimizing path tracing using noise reduction filters,” Winter School of Computer Graphics (WSCG) 1995 (1995), pp. 134-142. Bala et al. exploited an edge image to facilitate the filtering process. See, Bala et al., “Combining edges and points for interactive high-quality rendering,” ACM Trans. Graph. 22 (July 2003), 631-640. Xu and Pattanaik used a bilateral filter to remove MC noise. See, Xu and Pattanaik, “A novel Monte Carlo noise reduction operator,” IEEE Computer Graphics and Applications 25 (2005), 31-35. Egan et al. used frequency analysis to shear a filter for specific distributed effects such as motion blur and occlusion/shadowing. See, Egan et al., “Practical filtering for efficient ray-traced directional occlusion,” ACM Trans. Graph. 30, 6 (December 2011), 180:1-180:10; Egan et al., “Frequency analysis and sheared reconstruction for rendering motion blur,” ACM Trans. Graph. 28, 875 3, 1-13 (2009); Egan et al., “Frequency analysis and sheared filtering for shadow light fields of complex occluders,” ACM Trans. Graph. 30, 2 (April), 882 9:1-9:13 (2011). Mehta et al. used related analysis to derive simple formulas that set the variance of a screen-space Gaussian filter to target noise from specific effects. See, Mehta et al., “Axis aligned filtering for interactive sampled soft shadows,” ACM Trans. Graph. 31, 6 (November), 163:1-163:10 (2012); Mehta et al., “Axis-aligned filtering for interactive physically-based diffuse indirect lighting,” ACM Trans. Graph. 32, 4 (July), 96:1-933 96:12 (2013); Mehta et al., “Factored axis-aligned filtering for rendering multiple distribution effects,” ACMTrans. Graph. 33, 4 (July), 57:1-57:12 (2014). Some of these approaches use the analysis to adaptively position samples as well as conducting filtering and denoising operations.
Other filters use information in addition to or other than color. The additional information can include scene features (e.g., world positions, shading normals, texture values, etc.) which are computed by the MC renderer. Thus, they tend to generate higher-quality results compared to the color-based approaches described above. For example, McCool removed MC noise by using depths and normals to create a coherence map for an anisotropic diffusion filter. See, McCool, M. D., “Anisotropic diffusion for Monte Carlo noise reduction,” ACM Trans. Graph. 18, 2 (1999), 171-194. To efficiently render scenes with global illumination, Segovia et al. and Laine et al. used a geometry buffer. See, e.g., Segovia et al., “Non-interleaved Deferred Shading of Interleaved Sample Patterns,” Proceedings of the 21st ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware ACM, New York, N.Y., USA, GH '06, 53-60 (2006); Laine et al., “Incremental instant radiosity for real-time indirect illumination,” Proceedings of the 18th Eurographics Conference on Rendering Techniques, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, EGSR '07, 277-908 286 (2007). Meanwhile, to reduce global illumination noise, Dammertz et al. incorporated wavelet information into the bilateral filter. See, Dammertz et al., “Edge-avoiding A-trous wavelet transform for fast global illumination filtering,” Proceedings of High Performance Graphics 2010 (2010), pp. 67-75. Bauszat et al. used guided image filtering. See, Bauszat et al., “Guided image filtering for interactive high-quality global illumination,” Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering (EGSR)) 30, 4 (June 2011), 1361-1368. Shirley et al. used a depth buffer to handle depth of field and motion blur effects. See, Shirley et al., “A local image reconstruction algorithm for stochastic rendering,” Symposium on Interactive 3D Graphics and Games, ACM, New York, N.Y., 984 USA, I3D '11, 9-14 (2011). Chen et al. combined a depth map with sample variance to filter the noise from depth of field. See, Chen et al., “Efficient depth-of-field rendering with adaptive sampling and multiscale reconstruction,” Computer Graphics Forum 30, 6 (2011), 1667-1680. These methods are directed to a fixed set of distributed effects and are not applicable to general Monte Carlo rendering noise.
Hachisuka et al. performed adaptive sampling and reconstruction based on discontinuities in the multidimensional space. See, Hachisuka et al., “Multidimensional Adaptive Sampling and Reconstruction for Ray Tracing,” ACM Transactions on Graphics, Vol. 27, No. 3, Article 33, August 2008. Although this method handles general distributed effects, it suffers from the curse of dimensionality when a scene contains multiple effects.
To handle general MC noise using additional scene features, Sen and Darabi observed the need to vary a filter's feature weights across the image. See, Sen & Darabi, “On Filtering the Noise from the Random Parameters in Monte Carlo Rendering,” ACM Transactions on Graphics, Vol. 31, No. 3, March 2011. Specifically, they proposed to compute these weights using mutual information to approximate the functional dependencies between scene features and the random parameters. Li et al. used Stein's unbiased risk estimator (SURE) to estimate the appropriate spatial filter parameters in a cross-bilateral filter, while hard coding the weights of the remaining cross terms. See, Li et al, “SURE-based Optimization for Adaptive Sampling and Reconstruction,” ACM Transactions on Graphics, Vol. 31, No. 6, Article 194, November 2012. Rousselle et al. significantly improved upon this by using the SURE metric to select between three candidate cross non-local means filters that each weight color and features differently. See, Rousselle et al., “Robust denoising using feature and color information,” Computer Graphics Forum 32, 7, 121-130 (2013). Moon et al. compute a weighted local regression on a reduced feature space and evaluate the error for a discrete set of filter weights to select the best one. See, Moon et. al., “Adaptive Rendering Based on Weighted Local Regression,” ACM Transactions on Graphics (TOG) Volume 33 Issue 5, August 2014.
A significant drawback to the above approaches, which constitute the state of the art, is that they weight each filter term through either heuristic rules and/or an error metric which is quite noisy at low sampling rates. Thus, they are not able to robustly estimate the appropriate filter weights in challenging cases.
Neural networks have been used in computer graphics processing. Grzeszczuk used neural networks to create physically realistic animation. See, Grzeszczuk et al., “NeuroAnimator: Fast Neural Network Emulation and Control of Physics-Based Models,” Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, 887 ACM, New York, N.Y., USA, SIGGRAPH '98, 9-20 (1998). Nowrouzezahrai et al. used neural networks to predict per vertex visibility. See, Nowrouzezahrai et al. “Shadowing dynamic scenes with arbitrary BRDFs,” Computer Graphics Forum 28, 2, 249-258 (2009). Dachsbacher classified different visibility configurations using neural networks. See, Dachsbacher, C., “Analyzing Visibility Configurations,” IEEE Trans. Vis. And Comp. Graphics 17, 4, 475-476, 2011. Ren et al. used a neural network to model the radiance regression function to render indirect illumination of a fixed scene in real time. See, Ren et al., “Global Illumination with Radiance Regression Functions,” ACM Transactions on Graphics, Vol. 32, No. 4, Article 130, July 2013. Neural networks have also been directly trained on a set of noisy and clean patches and used to denoise natural images. However, methods applied to natural images are not applicable for rendered images, since they assume a single noise level for the entire image.
Jakob et al. have a method that, while not utilizing neural networks, performs learning through expectation maximization to find the appropriate parameters of a Gaussian mixture model to denoise photon maps. See, Jakob et al., “Progressive Expectation-Maximization for Hierarchical Volumetric Photon Mapping,” Eurographics Symposium and Rendering, Vol. 20, No. 4, 2011. Photon mapping has been applied to density estimation. Photon mapping is unrelated to Monte Carlo denoising, especially since no filtering is applied in the photon mapping. The Gaussian mixture models in this approach do not translate to Monte Carlo denoising.